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TITLE 



METHOD AND SYSTEM OF IMPLEMENTING REAL-TIME VIDEO - 
AUDIO INTERACTION BY DATA SYNCHRONIZATION 



BACKGROUND OP THE INVENTION 



5 Field of the Invention 

The present invention relates to a real-time 
video-audio interaction technology and in particular 
to a method of implementing real-time video-audio 
interaction by data synchronization in an Internet 
10 game. 

Description of the Related Art 

Generally speaking, the system architecture of an 
Internet game comprises an Internet game server and 
Internet game clients. The Internet game server 

15 receives and processes data from the Internet game 
clients to execute the Internet game. The Internet 
game clients interact with other Internet game clients 
and the Internet game. In other words, all the data 
required by the Internet game is sent to the Internet 

20 game server for data processing and data operation. 
Real-time video-audio interaction utilizes a. camera 
and microphones in an Internet game client to retrieve 
the real-time data and transmit the retrieved data 
through the Internet to other Internet game clients 

25 for interaction. 
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Three main technological difficulties limit 
Internet game systems' support of real-time video- 
audio interaction. First, current video display 
technology in an Internet game cannot display a real- 
5 time video picture if 3D animation occupies the 
display. Second, if the compression and transmission 
of video and audio data are executed solely in the 
Internet game server, the server becomes overloaded 
and bandwidth is compromised, affecting the speed of 
10 the Internet game. Third, the degree of video and 
audio data synchronization affects the result in real- 
time interaction. Hence, presently, real-time video- 
audio interaction is not supported in an Internet 
game. 

15 SUMMARY OP THE INVENTION 

Accordingly, an object of the invention is to 
provide a method of establishing an Internet 
transmission channel between Internet game clients, 
compressing and packaging real-time video-audio data 

20 in the Internet game clients, and transmitting the 
packaged data to other Internet game clients. The 
Internet game client that receives the transmission 
package, i.e. the recipient Internet game client, 
decodes the transmission package and applies a module 

25 simultaneously supporting real-time video and 3D 
animation to display the real-time video data when 3D 
animation occupies the display in an Internet game. 
In addition, the recipient Internet game client 
utilizes a sound card, cooperating with audio input or 
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output, to output the real-time audio data back. The 
module simultaneously supporting real-time video and 
3D animation outputs the real-time video data by 
integrating the real-time video data into game 
5 environments of the Internet game as texture mapping, 
accomplishing simultaneous display. 

To achieve the foregoing and other objects, the 
invention overcomes conventional real-time data 
interaction problems in an Internet game. In one 

10 embodiment, the inventive method first establishes an 
Internet transmission, channel between sender and 
recipient Internet game clients, without first 
connecting to an Internet game server. Next, the 
Internet game clients execute the Internet game and 

15 connect to the Internet game server. 

Then, real-time video and audio data are 
retrieved by the sender Internet game client, which 
then compresses/encodes the retrieved real-time video 
data into video data frames, compresses/encodes the 

20 real-time audio data into audio data packets, 
packaging both into a transmission package. The 
transmission package is then transmitted to the 
recipient Internet game client through the Internet 
transmission channel. A time stamp is attached in the 

25 transmission package for synchronizing the video and 
audio data. 

Finally, the recipient Internet game client 
decodes the transmission package into real-time video 
and audio data. The recipient Internet game client 
30 synchronizes the real-time video and audio data 



3 



FAX NO, OOOOOOOOOOOOOOOOOOOO P. 08 



Client's ref . : 92 022 

Pile : 0660-9881-OS/Pinal/Teresa/Steve 

according to the time stamp, outputs the decoded real- 
time video data in game environments, and outputs the 
decoded real-time audio data, achieving real-time 
video-audio interaction in the Internet game. 

5 BRIEF DESCRIPTION OP THE DRAWINGS 

The present invention can be more fully 
understood by reading the subsequent detailed 
description and examples with references made to the 
accompanying drawings, wherein; 
10 Fig. 1 is a flowchart of the method of 

implementing real-time video-audio interaction by data 
synchronization in an Internet game. 

Fig, 2 is a diagram of the system of implementing 
real-time video-audio interaction by data 
15 synchronization in an Internet game. 

Fig. 3 is a detailed diagram of the system of 
implementing real-time video-audio interaction by data 
synchronization in an Internet game. 

Fig, 4a, Fig. 4b are diagrams of the method of 
20 implementing real-time video-audio interaction by data 
synchronization in an Internet game, 

DETAILED DESCRIPTION OF THE INVENTION 

As summarized above, the present invention 
provides a system and method overcoming conventional 
25 real-time data interaction problems in an Internet 
game, first establishing an Internet transmission 
channel between Internet game clients without first 
connecting to an Internet game server. Next the 
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Internet game clients execute the Internet game and 
connect to the Internet game server. 

The Internet transmission channel is established 
as follows. First, the sender Internet game client 
5 assigns an Internet address of a recipient Internet 
game client directly or according to a directory which 
may include all the Internet addresses of possible 
recipient Internet game clients. The sender Internet 
game client then transmits a connection request to the 
10 recipient Internet game client. The recipient 
Internet game client establishes the Internet 
transmission channel in response to the connection 
request. 

Real-time video and audio data are retrieved in 
15 the sender Internet game client, which then 
compresses/encodes the retrieved real-time video data 
into video data frames, the real-time audio data into 
audio data packets, packaging both into a transmission 
package. The transmission package is then transmitted 
2 0 to the recipient Internet game client through the 
Internet transmission channel. 

If the bandwidth of the Internet transmission 
channel is not sufficient to transmit the real-time 
video and audio data, the real-time audio data takes 
2 5 priority over real-time video data. The advantage of 
considering audio data first is that audio data can be 
highly compressed. Thus, even if the bandwidth of the 
internet transmission channel is seriously deficient, 
the smooth of real-time audio data can be maintained, 
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achieving the basic requirement of real-time 
interaction. 

Finally, the recipient Internet game client 
decodes the transmission package into real-time video 
5 and audio data. The recipient Internet game client 
synchronizes the real-time video and audio data 
according to the time stamp, outputs the decoded real- 
time video and audio data in game environments. 

One important concern of video-audio data 

10 synchronization is maintaining correct time 
information for data display* If an Internet game 
client retrieves and displays real-time video-audio 
data, data synchronization is not a problem because 
the display time is synchronized to the system time of 

15 the Internet game client. If the Internet game client 
that retrieves real-time data is different from the 
display client, then the display client has to add the 
system time and the time stamp together as the display 
time . 

20 Conversely, because the transmission package 

considers the audio data first, there may be fewer 
transmitted video data frames than originally 
retrieved video data frames. The decompression of the 
transmission package may also drop some video data 

25 frames because of the data compression reference 
(IPB) . Thus, the mentioned time stamp and the amount 
of real-time video data must be applied to drop video 
data frames to achieve the expected result of video- 
audio data synchronization. The data synchronization 



6 



MAR- 11-2004 THU 13:44 



FAX NO. 00000000000000000000 P. 11 



Client's ref . ' 92022 

Pile : 0660-98Bi-US/Final/Teresa/steve 

can be processed before or after the video-audio data 
decompressing/decoding . 

Further, the invention provides a system of 
implementing real-time video-audio interaction by data 
5 synchronization in an Internet game. The inventive 
system includes an Internet game server and Internet 
game clients. The Internet game server executes an 
Internet game main program. The Internet game clients 
comprise interactive Internet game clients and an 

10 Internet transmission channel. The Internet game 
clients connect to the Internet game server, and the 
Internet transmission channel connects to the 
interactive Internet game clients, i.e. the sender and 
recipient Internet game clients. 

15 The sender Internet game client has a real-time 

data retriever, data encoder, transmission packager, 
and Internet sender. The real-time data retriever 
retrieves real-time video and audio data from the 
Internet game client. The data encoder 

20 compresses/encodes the real-time video data into video 
data frames, and the real-time audio data into audio 
data packets. The transmission packager then packages 
the encoded video and audio data packets into a 
transmission package. A time stamp is attached to the 

2 5 transmission package. The time stamp enables 

synchronization between the real-time video and audio 
data. The Internet sender transmits the transmission 
package to the recipient Internet game client through 
the Internet transmission channel. 
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Here, the Internet transmission channel is 
established according to the steps mentioned above. 
Again, if the bandwidth of the Internet transmission 
channel cannot accommodate the real-time audio and 
5 video data simultaneously, the real-time audio data 
takes priority over the real-time video data. 

The recipient Internet game client includes a 
data decoder and a video-audio playback system. The 
data decoder decodes the transmission package into 

10 real-time video and audio data. The video- audio 
playback system outputs the decoded video and audio 
data according the time stamp attached to the 
transmission package. Similarly, the video-audio 
playback system integrates the decoded real-time video 

15 data into game environments of the Internet game as 
texture mapping. The data is synchronized by adding 
the system time of the second internet game client and 
the time stamp together as the display time of the 
second real-time video and audio data. Data may also 

20 be synchronized comparing the time stamp to the amount 
of the second real-time video data and the number of 
frames dropped from the second real-time video data. 

Moreover, the invention provides a method of 
real-time video- audio interaction by data 

25 synchronization in an Internet game for application in 
an Internet game client. The Internet game client 
first establishes an Internet transmission channel to 
an external Internet game client without connecting to 
the Internet game server. The Internet game client 
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then connects to the Internet game server and executes 
the Internet game. 

Thereafter, real-time video and audio data are 
retrieved by a real-time data retriever and 
5 compressed/encoded into a transmission package. A 
time stamp is attached to the transmission package for 
data synchronization. The time stamp expresses the 
synchronous relationship between the real-time video 
and audio data. The transmission package is 

10 transmitted to the external Internet game client 
through the Internet transmission . channel . 

The Internet transmission channel is established 
by a specific procedure. The Internet game client 
assigns an Internet address to the external Internet 

15 game client directly or according to a directory which 
may include all the Internet addresses of possible 
external Internet game clients. The Internet game 
client then transmits a connection request to the 
external Internet game client. The external Internet 

20 game client establishes the Internet transmission 
channel in response to the connection request. If the 
bandwidth of the Internet transmission channel cannot 
support the real-time audio data and video data 
simultaneously, the real-time audio data takes 

25 priority over real-time video data. 

The Internet game client receives a transmission 
package from the external Internet game client through 
the Internet transmiss ion channel . The Internet game 
client decompresses/decodes the transmission package 

30 into real-time video and audio data. The Internet 
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game client then synchronizes the decoded video and 
audio data according to the time stamp attached to the 
transmission package, and outputs the decoded audio 
and video data in the Internet game. 
5 One embodiment of the invention is directed to a 

storage medium for storing a computer program, wherein 
the computer program, when applied to a computer 
system, implements real-time video and audio 
interaction by data synchronization between Internet 

10 game clients. The method may include the steps 
mentioned above. 

Another embodiment is directed to a computer 
system of an Internet game, executing an Internet game 
and having a storage medium for storing a computer 

15 program, wherein the computer program is applied to a 
computer system and implements real-time video-audio 
interaction by data synchronization between Internet 
game clients. The method may include the steps 
mentioned above, 

20 Fig. 1 is a flowchart of the method of 

implementing real-time video-audio interaction by data 
synchronization in an Internet game. The method first 
establishes an Internet transmission channel between 
clients without connecting to an Internet game server 

25 (step S100) . The Internet transmission channel can be 
established by any standard Internet transmission 
protocol, such as TCP/IP. The establishment of the 
Internet transmission channel is accomplished by 
certain steps. The sender Internet game client 

3 0 assigns an Internet address to a recipient Internet 
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game client directly or according to a directory which 
may include all Internet addressee of possible 
recipient Internet game clients. The sender Internet 
game client then transmits a connection request to the 
5 recipient Internet game client. The recipient 

Internet game client establishes the Internet 
transmission channel in response to the connection 
request . 

Next, the internet game clients connect to the 
10 internet game server for executing the internet game 
(step S101) . Real-time video and audio data are 
retrieved by the sender Internet game client (step 
S102) . The real-time video data may be retrieved by a 
video capture device, such as a camera. The real-time 
15 audio data may be retrieved by an audio capture 
device, such as a microphone. 

The sender Internet game client then 
compresses/encodes the real-time video data into video 
data frames, and compresses/encodes the real-time 
20 audio data into audio data packets (step S104) . 
Compression can be accomplished by a standard data 
compression format, such as H.263, MPEG 1/2/4, or 
G723. 

The video and audio data packets are then 
25 packaged into a transmission package (step S106) . 
Smooth audio data transmission and the synchronization 
between video and audio data are the chief concerns of 
the Internet real-time video-audio data transmission, 
therefore, the inventive method gives the highest 
30 transmission priority to audio data in the 
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transmission package, such that, if the bandwidth of 
the Internet transmission channel cannot 
simultaneously support the real-time video and audio 
data transmission, the audio data takes priority over 
5 video data. A time stamp is attached to the 
transmission package for synchronizing the video and 
audio data (step S107) . The time stamp attached to 
the transmission expresses the synchronous 
relationship between the real-time video and audio 

10 data, enabling synchronization between the real-time 
video and audio data. Thus, the inventive method 
guarantees smooth audio data transmission and accurate 
synchronization of video and audio data. 

The transmission package is then transmitted to 

15 the recipient Internet game client through the 
Internet transmission channel (step S108) . Finally, 
the recipient Internet game client decodes the 
transmission package into real-time video and audio 
data (step S110) , synchronizes the real-time video and 

20 audio data according to the time stamp (step Sill) , 
outputs the decoded real-time video data to the 
Internet game environments, and outputs the decoded 
real-time audio data (step S112) . The real-time 
interactive video data is integrated into the Internet 

25 game environments as texture mapping, to be 
simultaneously displayed in all the connected game 
environments as real-time video data. 

The data synchronization in the step Sill may be 
based on the system time of the second internet game 

30 client, adding the time stamp, as display time of the 



12 



MAR- 11-2004 THU 13:46 



FAX NO, 00000000000000000000 



P. 17 



Client's ref . : 92022 

Pile : 0660-9881-US/Final/Teresa/steve 

second real-time video and audio data. The data 
synchronization may base on the time stamp and refer 
to the amount of the second real-time video data to 
drop data frames of the second real-time video data. 
5 Thus, fewer video frames will be displayed in a fixed 
time period, that is, the display time of each video 
data frame is prolonged to allow video and audio data 
synchronization. 

The data synchronization of the step Sill can be 

10 processed before or after the step S110. If the data 
synchronization is processed before decoding, the 
video data frames may be dropped after decoding due to 
differences between decoders. if the data 

synchronization is processed after decoding, the 

15 dropped video frames are known, thus the number of 
dropped frames can be controlled. 

Fig. 2 is a diagram of the system of real-time 
video-audio interaction by data synchronization in an 
Internet game of the invention. In one embodiment, 

20 the inventive system includes an Internet game server 
20 and Internet game clients 24, 26, 28, 30. The 
Internet game server 20 executes an Internet game main 
program. The Internet game clients 24, 26, 28, 30, 
include interactive Internet game clients 24, 26, and 

25 an Internet transmission channel 32. The Internet 
game clients 24, 26, 28, 30, connect to the Internet 
game server 2 0 to establish an Internet game network 
22. The Internet transmission channel 32 connects to 
the interactive Internet game clients 24 and 26 

3 0 without connecting to the Internet game server 20. 
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Fig. 3 is a detailed diagram of the system of 
real-time video-audio interaction by data 
synchronization in an Internet game. The sender 
Internet game client 300 has a real-time data 
5 retriever 3 02, a data encoder 304, a transmission 
packager 306, and an Internet sender 308. The real- 
time data retriever 3 02 retrieves real-time video and 
audio data from the Internet game client 300, The 
real-time data retriever 302 may comprise separate 

10 devices retrieving and processing video and audio data 
separately or an integrated device. The real-time 
data retriever 302 can be any data capture device, 
such as a camera or microphone. 

The data encoder 3 04 compresses/encodes the real- 

15 time video data into video data frames, and the real- 
time audio data into audio data packets. The data 
encoder 304 may comprise separate devices to 
compress/encode video and audio data separately or an 
integrated device. 

20 The transmission packager 306 assigns the audio 

frames highest transmission priority and packages the 
video frames and audio data packets into a 
transmission package. A time stamp is attached to the 
transmission package. The time stamp expresses the 

25 synchronous relationship between the video and audio 
data, that is, the time information of producing video 
and audio. The Internet sender 3 08 transmits the 
transmission package to the recipient Internet game 
client 32 0 through the Internet transmission channel 

30 310. As an example, the transmission packager 306, 
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the Internet sender 308, and the Internet transmission 
channel 310 can be implemented by programming into 
computer-executive modules in program languages to 
achieve the operations described. 
5 The recipient Internet game client 320 includes a 

data decoder 322 and a video -audio playback system 
324. The data decoder 322 decodes the transmission 
package into real-time video and audio data. Here, 
the data decoder 322 can comprise separate devices or 

10 be an integrated device. The data encoder 304 and the 
data decoder 322 may be implemented by programming 
into computer-executive modules in program languages 
to achieve the mentioned functions. 

The video- audio playback system 324 synchronizes 

15 the decoded video and audio data according to the 
attached time stamp. The video-audio playback system 
324 integrates the decoded real-time video data into 
game environment of the Internet game as texture 
mapping. The video-audio playback system 324 outputs 

20 the real-time audio data by a sound card with audio 
input or output. The real-time video-audio 

interaction between the Internet game clients is bi- 
directional, that is # the Internet game clients for 
real-time video -audio interaction are provided with 

25 both sender and recipient functions. All components 
and functions of both sender and recipient are 
provided in the sender Internet game client 300 and 
the recipient Internet game client 320. 

Figs. 4a, Fig. 4b are diagrams of the method of 

30 implementing real-time video- audio interaction by data 
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synchronization in an Internet game. An Internet 
transmission channel 50 is established for 
transmission. The camera 40 retrieves real-time video 
data and microphone 42 retrieves real-time audio data. 
5 The data encoder 44, 46, compresses/encodes the 

real-time video data into video data frames, 440, 442, 
444, 446, P frames. The data encoder 44, 46, also 
compresses/encodes the real-time audio data into audio 
data packets 460, 462, 464, L packets. The audio data 

10 packet takes priority over the video data frames. The 
video data frames and audio data packets are packaged 
into a transmission package 48. In other words, the 
audio data frame 460 is packaged first into the 
transmission package in one fixed time period and the 

15 video data frames 44 0, 442 are then packaged according 
to remaining bandwidth. If only N video data frames 
are packaged into the transmission package, then (P-N) 
video data frames are not. A time stamp is attached 
to the transmission package. The time stamp expresses 

20 the synchronous relationship between the video and 
audio data. 

Through the Internet transmission channel 50, the 
sender Internet client transmits the transmission 
package 48 to the recipient Internet game client. The 

25 data decoders 52 and 54 decode the transmission 
package 48 into real-time video data 520 to 522 and 
real-time audio data 540. The Internet transmission 
may drop some video frames, only M video frames are 
received (M<=N) . Obviously, the number of video data 

3 0 frames 440 to 442 is equal to or larger than the 
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number of video data frames 520 to 522. The video 
decompression method, such as IPB mode, may also cause 
video data frame drop such that only K (K<=M) video 
data frames are decompressed. Audio frames 540 are 
5 decompressed completely. Thus, the first package 
mechanism ensures smooth audio transmission and 
accurate synchronization of the video and audio data. 

The video-audio playback systems 56 and 58, 
display the decoded video data by simultaneous support 

10 of real-time video and 3D animation, outputting real- 
time video data to game environments as texture 
mapping. The video-audio displayers 56 and 58, may 
synchronize data by adding the system time of the 
second internet game client to the time stamp and 

15 generating the display time of the second real-time 
video and audio data. Alternately, the video-audio 
displayers 56 and 58 may display synchronized data by 
comparing the time stamp to the number of frames 
dropped by the second real-time video data. The data 

20 synchronization can be accomplished before or after 
the data has been decoded by the decoders 52 and 54. 
The video-audio playback systems 56, 58, output audio 
data via a sound card with audio input or output 58, 
achieving real-time video-audio interaction. 

25 The methods and systems provided by the present 

invention can be independent from an Internet game 
system by establishing an Internet transmission 
channel between Internet game clients. In addition, 
the invention applies the first audio data package 

30 mechanism and an attached time stamp to achieve smooth 
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audio transmission and accurate synchronization of the 
video and audio data, accomplishing real-time video- 
audio interaction by data synchronization in an 
Internet game. The invention enhances the reality of 
5 the Internet game without further burdening the 
Internet game server, resolving problems of the 
existing technology. 

It will be appreciated from the foregoing 
description that the system and method described 

10 herein provide a dynamic and robust solution to real- 
time video-audio interaction problems, If, for 
example, an Internet game client changes data capture 
devices, the system and method of the present 
invention can be revised accordingly. 

15 The methods and system of the present invention, 

or certain aspects or portions thereof, may take the 
form of program code (i.e., instructions) embodied in 
tangible media, such as floppy diskettes, CD-ROMS, 
hard drives, or any other machine -readable storage 

20 medium, wherein, when the program code is loaded into 
and executed by a machine, such as a computer, the 
machine becomes an apparatus for practicing the 
invention; The methods and apparatus of the present 
invention may also be embodied in the form of program 

25 code transmitted over some transmission medium, such 
as electrical wiring or cabling, through fiber optics, 
or via any other form of transmission, wherein, when 
the program code is received and loaded into and 
executed by a machine, such as a computer, the machine 

30 becomes an apparatus for practicing the invention. 
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When implemented on a general -purpose processor, the 
program code combines with the processor to provide a 
unique apparatus that operates analogously to specific 
logic circuits. 
5 While the invention has been described by way of 

example and in terms of the preferred embodiments, it 
is to be understood that the invention is not limited 
to the disclosed embodiments. To the contrary, it is 
intended to cover various modifications and similar 
10 arrangements (as would be apparent to those skilled in 
the art) - Therefore, the scope of the appended claims 
should be accorded the broadest interpretation so as 
to encompass all such modifications and similar 
arrangements. 
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